Efficient and Robust Compressed Sensing using 
High-Quality Expander Graphs 



Sina Jafarpour Weiyu Xu Babak Hassibi Robert Calderbank 

Computer Science Electrical Engineering Electrical Engineering Electrical Engineering 

Princeton University California Institute of Technology California Institute of Technology Princeton University 

sina@cs.princeton.edu weiyu@caltech.edu hassibi@caltech.edu calderbk@math.princeton.edu 



Abstract — Expander graphs have been recently proposed to 
construct efficient compressed sensing algorithms. In particular, 
it has been shown that any n -dimensional vector that is k- 
sparse (with k <§C n) can be fully recovered using O(fclog^) 
measurements and only O(fclogn) simple recovery iterations. In 
this paper we improve upon this result by considering expander 
graphs with expansion coefficient beyond | and show that, with 
the same number of measurements, only 0(k) recovery iterations 
are required, which is a significant improvement when n is 
large. In fact, full recovery can be accomplished by at most 
2k very simple iterations. The number of iterations can be 
made arbitrarily close to k, and the recovery algorithm can be 
implemented very efficiently using a simple binary search tree. 
We also show that by tolerating a small penalty on the number of 
measurements, and not on the number of recovery iterations, one 
can use the efficient construction of a family of expander graphs 
to come up with explicit measurement matrices for this method. 
We compare our result with other recently developed expander- 
graph-based methods and argue that it compares favorably both 
in terms of the number of required measurements and in terms 
of the recovery time complexity. Finally we will show how our 
analysis extends to give a robust algorithm that finds the position 
and sign of the k significant elements of an almost fc-sparse signal 
and then, using very simple optimization techniques, finds in 
sublinear time a k -sparse signal which approximates the original 
signal with very high precision. 

I. Introduction 

The goal of compressive sampling or compressed sensing is 
to replace the conventional sampling and reconstruction oper- 
ations with a more general combination of linear measurement 
and optimization in order to acquire certain kinds of signals 
at a rate significantly below Nyquist. Formally suppose we 
have a signal x which is sparse. We can model x as a n 
dimensional vector that has at most k non-zero components . 
We desire to find an m x n matrix A such that m , the number 
of measurements, becomes as small as possible ( and can be 
efficiently stored ) and x can be recovered efficiently from 
y = Ax. 

The originally approach was through the use of dense 
random matrices and random projections. It has been shown 
that if the matrix A has some restricted isometry property 
(RIP-2), that is, it almost preserves the Euclidean norm of 
all 3k — sparse vectors, then A can be used in compressed 
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sensing and the decoding can be accomplished using linear 
programming and convex programming methods [17]. This 
is a geometric approach based on linear and quadratic op- 
timization, and [19] showed that the RIP — 2 property is 
a direct consequence of the Johnson-Linderstrauss lemma 
[20] so that many dense random matrices will satisfy this 
property. However, the problem in practice is that the linear 
and quadratic programming algorithms have cubic complexity 
in n and become really inefficient, as n becomes very large; 
furthermore, in order to store the whole matrix in memory we 
still need 0(m x n) which is inefficient too. 

Following [l]-[3], [22], [23], we will show how random 
dense matrices can be replaced by the adjacency matrix of a 
high quality family of expander graphs, thereby reducing the 
space complexity of matrix storage and, more important, the 
recovery time complexity to a few very simple iterations. The 
main idea is that we study expander graphs with expansion 
coefficient beyond the | that was considered in [1], [2]. 

The remainder of the paper is organized as follows. In 
Section II we review the previous results from [1], [2]. In 
Section III, following the geometric approach of [3], we 
establish that the adjacency matrix of the expander graphs 
satisfies a certain Restricted Isometry Property for Manhattan 
distance between sparse signals. Using this property, or via a 
more direct alternative approach, we show how the recovery 
task becomes much easier. In Section IV we generalize the 
algorithm of [1], [2] to expander graphs with expansion 
coefficient beyond |. The key difference is that now the 
progress in each iteration is proportional to log n, as opposed 
to a constant in [1], [2], and so the time complexity is 
reduced from O(klogn) to 0(k). We then describe how the 
algorithm can be implemented using simple data structures 
very efficiently and show that explicit constructions of the 
expander graphs impose only a small penalty in terms of the 
number of measurements, and not the number of iterations, 
the recovery algorithm requires. We also compare our result 
to previous results based on random projections and to other 
approaches using the adjacency matrices of expander graphs. 
In section V generalize the analysis to a family of almost k- 
sparse signals; (after a few very simple iterations) the robust 
recovery algorithm proposed in [1] empowered with high- 
quality expander graphs finds the position and the sign of 
the k significant elements of an almost /c-sparse signal. Given 



this information, we then show how the restricted isometry 
property of the expander graphs lets us use very efficient 
sub-linear optimization methods to find a fc-sparse signal that 
approximates the original signal very efficiently. Section VI 
concludes the paper. 

II. Previous result: O(fclogn) recovery 

A. Basic Definitions 

Xu and Hassibi [1] proposed a new scheme for compressed 
sensing with deterministic recovery guarantees based on com- 
binatorial structures called unbalanced expander graphs: 

Definition 1 (Bipartite Expander Graph, Informally): An 
expander graph [5] E is a d regular graph with v vertices, 
such that: 

1) E is sparse (ideally d is much smaller than v). 

2) E is " well connected". 

Various formal definitions of the second property define the 
various types of expander graphs. The expander graph used in 
[1], [2] which has suitable properties for compressed sensing 
is the "vertex expander" or "unbalanced expander" bipartite 
graph: 

Definition 2 (Unbalanced Bipartite —Expander Graph): 
A bipartite left regular graph with n variable nodes, m parity 
check nodes and left degree d will be (an, expander 
graph, for < a < |, if for every subset of variable nodes 
V with cardinality |V| < an, the number of neighbors 
connected to V, denoted by N(V) is strictly larger than \dV, 
i.e, \N(V) > §d|V|. 

Using the probabilistic method, Pinsker and Bassylago [7] 
showed the existence of | —expander graphs and they showed 
that any random left-regular bipartite graph will, with very 
high probability, be an expander graph. Then Capalbo et al. 
gave an explicit construction for these expander graphs. 

Theorem 1: Let < r < 1 be a fixed constant. Then for 
large enough n there exists a (an, |ci) expander graph E with 
n variable nodes and - parity check nodes with constant left 
degree (not growing with n) d and some < a < 1. Further- 
more, the explicit zig-zag construction can deterministically 
construct the expander graph. 

Using Hoeffding's inequality and Chernoff bounds Xu and 
Hassibi [2] showed the following theorem. 

Theorem 2: For any k, if n is large enough, there exists a 
left regular bipartite graph with left degree d for some number 
d, which is (k, expander graph with m = O(klogn) 
parity check nodes. 

B. Recovery Algorithm 

Suppose x is the original n dimensional /c-sparse signal, and 
the adjacency matrix of a (an, |d) expander graph is used as 
the measurement matrix for the compressed sensing. We are 
given y = Ax and we want to recover x. Xu and Hassibi [1] 
proposed the following algorithm: 

In the above algorithm the gap is defined as follows. 

Definition 3 (gap): Let x be the original signal and y = 
Ax. Furthermore, let x be our estimate for x. For each value 



Algorithm 1 Left Degree Dependent Signal Recovery algo- 
rithm 

1) Initialize x — 0„ x i- 

2) IF y = Ax output x and exit. 

3) ELSE find a variable node say Xj such that more than half 
of the measurements it participate in, have identical gap 
9- 

4) set Xj <— Xj + g, and go to 2. 



yi we define a gap g t as: 

n 

9i Vi ^ y , A-ij Xj . 

Xu and Hassibi [1] proved the following theorem that 
bounds the number of steps required by the algorithm to 
recover i. 

Theorem 3: Suppose A is the adjacency matrix of an ex- 
pander graph satisfying Definition 2, and x is an n dimensional 
k sparse signal (with k < g ^), and y = Ax. Then Algorithm 
1 will always find a signal x which is k sparse and for which 
Ax = y. Furthermore, the algorithm requires at most 0(kd) 
iterations, where k is the sparsity level of the signal and d is 
the left side degree of the expander graph. 

Let us now consider the consequences of the above Theorem 
for the expander graphs in Theorems 1 and 2. In Theorem 1 
the sparsity can grow proportional to n (since k < ^) and 
the algorithm will be extremely fast; the algorithm requires 
0(kd) iterations and since d is a constant independent of n, 
the number of iterations will be 0(k). We also clearly need 
0(n) measurements. 

In Theorem 2 the sparsity level k is fixed (does not 
grow with n) and the number of measurements needs to 
be O(fclogn), which is desired. Once more the number 
of required iterations is 0(kd). However, in this case Xu 
and Hassibi showed the following negative result for (k, |d) 
expander graphs. 

Theorem 4: Consider a bipartite graph with n variable 
nodes and m measurement nodes, and assume that the graph 
is a (k, jd) expander graph with regular left degree d. Then 
if m — O(klogn) we have d = il(logn). 

This theorem implies that for a (k, |d) expander graph, the 
recovery algorithm needs O(fclogn) iterations. The main con- 
tribution of the current paper is that the number of iterations 
can be reduced to 0(k). The key idea is to use expanders with 
expander coefficient beyond |. 

Remark Theorem 3 does not imply the full recovery of the 
sparse signal. It only states that the output of the recovery 
algorithm will be a k sparse signal x such that Ax = Ax 
where x is the original signal. However, in the next section 
we show how an interesting property of the expander graphs 
called the RIP — 1 property, implies the full recovery. We 
also give a direct proof by showing that the null-space of the 
adjacency matrix of an expander graph cannot be "too sparse". 



III. Expander Codes, RIP-1 Property, and 
Full-Recovery Principle 

A. Expander Codes 

Compressed Sensing has many properties in common with 
coding theory. The recovery algorithm is similar to the decod- 
ing algorithms of error correcting codes but over R m instead 
of a finite field. As a result, several methods from coding 
theory have been generalized to derive compressed sensing 
algorithms. Among these methods are the generalization of 
Reed-Solomon codes by Tarokh [9], and very recent results 
by Calderbank, et al [8], which are based on second order 
Reed-Muller codes, and Parvaresh et al [10], based on list 
decoding. 

In 1996, Sipser and Spielman [11] used expander graphs 
to build a family of linear error-correcting codes with linear 
decoding time complexity. These codes belong to class of error 
correcting codes called Low Density Parity Check (LDPC) 
Codes. The work done by Xu and Hassibi is a generalization 
of these expander codes to compressed sensing. Feldman et 
al [12] suggested a way of decoding expander codes using 
linear programming, and linear programming is the usual 
recovery algorithm in compressed sensing. This leads to a 
better understanding of compressed sensing using expander 
graphs and a very different geometric perspective on the 
problem. 

B. Norm one Restricted Isometry Property 

The standard Restricted Isometry Property, is an impor- 
tant condition that enables compressed sensing using random 
projections. Intuitively, it says that the measurement almost 
preserves the euclidean distance between any two sufficiently 
sparse vectors. This property implies that recovery using l\ 
minimization is possible if a random projection is used for 
measurement. Indyk and Berinde in [3] showed that expander 
graphs satisfy a very similar property called RIP-1 which states 
that if the adjacency matrix of an expander graph is used for 
measurement, then the Manhattan (li) distance between two 
sufficiently sparse signals will be preserved by measurement. 
They used this property to prove that ^-minimization is still 
possible in this case. However, we will show in this section 
how RIP-1 can guarantee that the algorithm described will 
have full recovery. 

Following [3], we will show that the RIP-1 property can be 
derived from the expansion property and will guarantee that if 
x is the original fc-sparse signal, then no recovery algorithm 
can output a fc-sparse signal x such that i^i but Ax = Ax. 

We begin with the definition of expander graphs with 
expansion coefficient 1 — e, bearing in mind that we will be 
interested in 1 — e > f . 

Definition 4 (Unbalanced Expander Graph): A (/, 1 — e)- 
unbalanced bipartite expander graph is a bipartite graph E = 
(A, B), \ A\ = n, \B\ — to, where A is the set of variable nodes 
and B is the set of parity nodes, with regular left degree d 
such that for any X C A, if \X\ < I then the set of neighbors 
N(X) of X has size N(X) > (1 - e)d\X\. 
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(k,e) vertex expander 
Fig. 1. (k,e) vertex expander graph 

The following claim can be derived using the Chemoff 
bounds [3] 1 : 

Claim 1: for any | > I > 1 , e > there exists a (1,1 — e) 
expander with left degree: 

and right set size: 




Lemma 1 (RIP-1 property of the expander graphs): Let 
A m xn be the adjacency matrix of a (fc, e) expander graph E, 
then for any fc-sparse vector x S lZ n we have: 

(1 -2e) d \\x\\i < \\Ax\lx <d \\x\\i (1) 

Proof: The upper bound is trivial using the triangle 
inequality, so we only prove the lower bound: 

The left side inequality is not influenced by changing the 
position of the coordinates of x, so we can assume that they 
are in a non-increasing order: |jei| > \x2\ > ... > \x n \. Let 
(xi,yj) be the edge e that connects x.- L to yj. Define E2 = 
{(xi,yj) : 3k < i : (xk,Uj) G E}. Intuitively E2 is the set of 
the collision edges. Let Ti = {e : 3j s.t : e = (xi,yj),e S 
E 2 } and a, = \T Z \. 

Clearly a\ = 0; moreover by the expansion property of the 
graph we have: for all fc' < fc : < edk', and finally since 
the graph is fc-sparse we know that for all fc" > fc : Xk" = 0. 
Hence 

(x l ,y,)EE 2 i=l 

=J2 a i(\ x i \ - Nt+ll) 
i<k 

<^2edi(\xi\ - \x i+1 \) 

i<k 

\ x i\ ed 

i<k 

=ed \x\ 1 . 

'This claim is also used in the expander codes construction 



Now the triangle inequality implies: 

|^4ar|i>| 
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=(l-2e) d |M|i. 
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C. Full Recovery 

The full recovery property now follows immediately from 
Lemma 1. 

Theorem 5 (Full recovery): Suppose A mxn is the adja- 
cency matrix of a (3fc, 1 — e) expander graph. And x\ is a 
k — sparse and x 2 is a 2/c-sparse vector, such that Ax\ = Ax 2 
then x\ = x 2 . 

Proof: Let z = x\ — x 2 . Since x\ is k — sparse and £2 
is 2fc-sparse, z is 3fc-sparse 2 . By Lemma 1 we have: 

Iki - x 2 \\i < (1 _ 1 2e) j ||Aci - Ax 2 \\i = 0, 

hence Xi = x 2 . ■ 
Note that the proof of the above theorem essentially says 
that the adjacency matrix of a (3k, 1 - e) expander graph does 
not have a null vector that is 3k sparse. We will also give a 
direct proof of this result (which does not appeal to RIP-1) 
since it gives a flavor of the arguments to come. 

Lemma 2 (Null space of A): Suppose A mxn is the adja- 
cency matrix of a (3k, 1 — e) expander graph with e < \. Then 
any nonzero vector in the null space of A, i.e., any z^O such 
that Az — 0, has more than 3k nonzero entries. 

Proof: Define S to be the support set of z. Suppose that z 
has at most 3k nonzero entries, i.e., that |<S| < 3k. Then from 
the expansion property we have that N(S) > (1 — e)d|«S|. 
Partitioning the set N(S) into the two disjoint sets Ni(S), 
consisting of those nodes in N(S) that are connected to a 
single node in S, and N > i(S), consisting of those nodes 
in N(S) that are connected to more than a single node in 
S, we may write N^S) + N>i(<S) > (1 - e)d|«S|. Further- 
more, counting the edges connecting S and N(S), we have 
\Ni(S)\ + 2|JV>i(5)| < d\S\. Combining these latter two 
inequalities yields |7Vi(6>)| > (1 — 2e)rf|<S| > 0. This implies 
that there is at least one nonzero element in z that participates 
in only one equation of y = Az. However, this contradicts the 
fact that Az = and so z must have more than 3k nonzero 
entries. ■ 

IV. Our results: Efficient Full Recovery 

A. Efficient O(klogn) measurement with 0(k) iteration re- 
covery 

In this section, we show the general unbalanced bipartite 
expander graphs introduced in Definition 4 work much better 
than |-expanders, in the sense that it gives the measurement 

2 1 1 1 1 o < ||xi||o + ||z2||o = 3fe 



size O(klogn) which is up to a constant the optimum mea- 
surement size, and simultaneously yields a recovery algorithm 
which needs only 0(k) simple iterations. 

Before proving the result, we introduce some notations used 
in the recovery algorithm and in the proof. 

Definition 5 (gap): Recall the definition of the gap from 
Definition 3. At each iteration t, let G t be the support 3 of the 
gaps vector at iteration t : 

n 

G t = support (g t ) = {i\yi ^^A^Xj}. 

Definition 6: At each iteration t, we define S t an indicator 
of the difference between the estimate x and x : 

S t = support (x — x) — {j : Xj =^ Xj}. 

Now we are ready to state the main result: 

Theorem 6 (Efficient and Certain Compressive Sampling ): 
Let e < j and suppose A mxn , as defined in definition 4 

where m — O ^ k l °§ " ^ , be the adjacency matrix of a (3fc, e) 
expander graph. If we use A as the measurement matrix 
in compressed sensing of fc-sparse signals, the following 
algorithm 2 will recover the original signal k sparse signal 
x from its measured sketch y = Ax with certainty using at 
most 0(k) simple iterations . 



Algorithm 2 Our 0(k) iteration, recovery algorithm 

1) Initialize x = nx i- 

2) IF y = Ax output x and exit. 

3) ELSE find a variable node say xj such that at least (1 — 
2e) d of the measurements it participate in, have identical 
gap 9- 

4) set Xj <— Xj + g, and go to 2. 



The proof is virtually identical to that of [1], except that we 
consider a general 1 — e expander, rather than a | -expander, 
and consists of the following lemmas. 

* The algorithm never gets stuck, and one can always find a 
coordinate j such that Xj is connected to at least (1 — 2e)d 
parity nodes with identical gaps. 

• With certainty the algorithm will stop after at most 2k 
rounds. Furthermore, by choosing e small enough the 
number of iterations can become arbitrarily close to k. 

Lemma 3 (progress): Suppose at each iteration t, St = {j : 
Xj ^ Xj}. If \St\ < 2k then always there exists a variable node 
Xj such that at least (1 — 2e)d of its neighbor check nodes have 
the same gap g. 

Proof: we will prove that there exists a coordinate j, such 
that Xj is connected to at least (1 — 2e)d check nodes uniquely, 
in other words no other variable node is connected to these 
nodes. This immediately implies the lemma. 
Since \St\ < 2k by the expansion property of the graph 
N(S t ) > (1 — e)d\S t \. Now we are going to count the 

3 set of nonzero elements 




Fig. 2. Progress Lemma 

neighbors of St in two ways. Figure 2 shows the progress 
lemma. 

We partition the set N(S t ) into two disjoint sets: 

• Ni(St): The vertices in N(St) that are connected only 
to one vertex in St. 

• N > i(S t )'. The other vertices (that are connected to more 
than one vertex in St). 

By double counting the number of edges between variable 
nodes and check nodes we have: 

|iVi(5 t )| + |JV>i($)| = \N(S t )\ > (1 - e)d\S t \ 
\N 1 (St)\ + 2\N >1 (S t )\ < #edges between S t ,N(S t ) = d\S t \ 
This gives 

|JV>!(5 t )| <ed\S t \, 

hence 

|JVi(S t )| > (l-2e)d\S t \, 

so by the pigeonhole principle, at least one of the variable 
nodes in St must be connected uniquely to at least (1 — 2e)d 
check nodes. ■ 
Lemma 4 (gap elimination): At each step t if \St\ < 2k 
then |G t+1 | < |G t |-(l-4e)d 

Proof: By the previous lemma, if | St \ < 2k, there always 
exists a node Xj that is connected to at least (1 — 2e)d nodes 
with identical nonzero gap , and hence to at most 2ed nodes 
possibly with zero gaps. Setting the value of this variable node 
to zero, sets the gaps on these uniquely connected neighbors of 
Xj to zero, but it may make some zero gaps on the remaining 
2ed neighbors non-zero. So at least (1 — 2e)d coordinates of 
Gt will become zero, and at most 2ed its zero coordinates may 
become non-zero. Hence 

|G t+1 | < |G t |-(l-2e)d + 2ed=|G t |-(l-4e)d. (6) 

■ 

Remark: The key to accelerating the algorithm is the above 
Lemma. For a | expander, e = j and so |Gt+i| < \Gt\, 
which only guarantees that |Gf+i| is reduced by a constant 
number. However, when e < 4, we have |Gt+i| < \Gt\ — 
(1 — 4e)d, which means that |Gt+i| is guaranteed to decrease 
proportionally to d. Since d — f2(logn), we save a factor of 
log n. 

Lemma 5 (preservation): At each step t if \St\ < 2k, after 
running the algorithm we have |<St+i| < 2k. 



Proof: Since at each step we are only changing one 
coordinate of x, we have |i>t+i| = \St\ + 1, so we only need 
to prove that St+i ^ 2k. 

Suppose for a contradiction that |5t+i| = 2k, and partition 
N(St+i) into two disjoint sets: 

1) Ni(St+i): The vertices in N(St+i) that are connected 
only to one vertex in St+i. 

2) N > i(St+x). The other vertices (that are connected to 
more than one vertex in S t +i). 

The argument is similar to that given above; by double 
counting the number of vertices in Nx(St+i), N > i(St+%) one 
can show that 

\Ni(S t+ i)\ > (l-2e) d2k 

Now we have the following facts: 

. |iVi(5 f+ i)| < |G t+ i| : Coordinates in N^St+i) are 
connected uniquely to coordinates in St+i, hence each 
coordinate in N\(St+i) has non-zero gap. 

• |Gt+i| < |Gi|: gap elimination from Lemma 4. 

• I Gi| < kd: x, x differ in at most k coordinates, so 

can differ in at most kd coordinates. 
As a result we have: 

(1 - 2e)2 dk < |iVi(5 t+ i)| < |G t+1 | < |Gi| < kd (8) 

This implies e > h which contradicts the assumption e < \. 

■ 

Proof of the Theorem 6: Preservation (Lemma 5) and 
progress (Lemma 3) together immediately imply that the 
algorithm will never get stuck. Also by Lemma 4 we had 
shown that |Gi| < kd and |G t+ i| < \G t \ - (1 - 4e)d. Hence 
after at most T = jzt^r t steps we will have | Gt | = and this 
together with the preservation lemma implies that we have 
discovered a signal x such that x is 2fc-sparse and Ax = y. 
Now since we had used a (3fc,e) expander, the full recovery 
property (Theorem 5) guarantees the recovery of the original 
signal. 

Note that we have to choose e < j, and as an example, by 
setting e = g the recovery needs at most 2k iterations. ■ 

Remark: The condition e < \ in the theorem is necessary. 
Even e = \ leads to a | expander graph (Definition 2), which 
needs O(fclogn) iterations. 

B. Explicit Construction of Expander Graphs 

In the definition of the expander graphs (Definition 4), we 
noted that probabilistic methods prove that such expander 
graphs exist and furthermore, that any random graph will, with 
high probability, be an expander graph. Hence, in practice it 
may be sufficient to use random graphs instead of expander 
graphs. 

Though, there is no efficient explicit construction for the 
expander graphs of Definition 4, there exists an explicit 
construction for a class of expander graphs which are very 
close to the optimum expanders of Definition 4. Recently [13], 
Guruswami et al based on Parvaresh-Vardy codes [14], proved 
the following theorem: 



Theorem 7 (Explicit Construction of expander graphs): 
For any constant a > 0, and any n, k, e > 0, there exists a 
(fc, 1 — e) expander graph with left degree: 



and number of right side vertices: 

TO = 0(d 2 k 1+a ) 

which has an efficient deterministic explicit construction. 
Since our previous analysis was only based on the expansion 
property, which does not change in this case, a similar result 
holds if we use these expanders. For instance by letting a = 1 
and e = | we will have an explicit expander construction with 
d = O ((logn) 2 ) and so we just need m — 0(k 2 d 2 ) number 
of measurements, and at most 2k number of iterations in the 
recovery algorithm. 

C. Comparison with the recent unified geometric- 
combinatorial approach 

We will compare our result with a very recent result by 
Indyk et al [22]. Their result unifies Indyk's previous work 
which was based on randomness extractors [23] and a combi- 
natorial algorithm with another approach to the RIP-1 property 
of Indyk-Berinde [3] which is based on geometric convex opti- 
mization methods and suggests a recursive recovery algorithm 
which takes m' = 0(7nlogn) = 0(klog 2 n) sketch measure- 
ments and needs a recovery time 0(m log 2 n) = 0(fclog 3 n). 

By comparison, our recovery algorithm is a simple itera- 
tive algorithm, that needs 0(k\ogn) sketch measurements. 
Our decoding algorithm consists of at most 2k very simple 
iterations. Each iteration can be implemented very efficiently 
(see [1] ) since the adjacency matrix of the expander graph 
is sparse with all entries or 1. One naive way to do that 
is by using balanced binary search trees 4 . As shown before, 
initially \G\\ < kd so we can build the tree efficiently in 
O(kdlogd). Now by gap elimination (Lemma 4), although at 
each iteration some nodes are going to be deleted from the 
tree and some new nods are added, the size of the tree does 
not grow, so all the updates can be done in O(dlogd) . As 
a result, we have a O(kdlogd) preprocessing step, and 2k 
iterations each taking O(dlogd). So this naive approach has 
overall time complexity O(kdlogd) = 0(k logn log logn). 
This can even be improved by using better data structures. 

V. Approximately sparse signals and robust 

RECOVERY 

In this section we will show how the analysis using high- 
quality expander graphs that we proposed in the previous 
section can be used to show that the robust recovery algorithm 
in [1] can be done more efficiently in terms of the sketch size 
and recovery time for a family of almost fc-sparse signals. 
With this analysis we will show that the algorithm will only 
need O(fclogn) measurements. Explicit constructions for the 

4 such as red-black trees 



sketch matrix exist and the recovery consists of two simple 
sub-linear steps. First, the combinatorial iterative algorithm in 
[1] , which is now empowered with the high-quality expander 
sketches, can be used to find the position and the sign of the 
k largest elements of the signal x. Using an analysis similar 
to the analysis in section IV we will show that the algorithm 
needs only O(fc) iterations, and similar to the previous section, 
each iteration can be done efficiently using a variation of 
red-black trees and will have time complexity 0(d\ogd) = 
0(logn log logn). Then restricting to the position of the k 
largest elements, we will use a robust theorem in expander 
graphs to show that simple optimization methods that are now 
restricted on k dimensional vectors can be used to recover a k 
sparse signal that approximates the original signal with very 
high precision. In summary, both the combinatorial part and 
the optimization part require sub linear time complexity so 
the overall algorithm needs sub linear recovery time and will 
output a fc-sparse signal very close to the original signal. 

Before presenting the algorithm we will define precisely 
precisely what we mean for a signal to be almost k sparse. 

Definition 7 (almost k-sparse signal): A signal x £ R n is 
said to be almost fc-sparse iff it has at most k large elements 
and the remaining elements are very close to zero and have 
very low magnitude. In other words, the entries of the near- 
zero level in the signal vector are near-zero elements taking 
values from the set [—A, A] while the significant level of entries 
take values from the set S = {x : \L — A| < \x\ < \L + 
A}. By the definition of the almost sparsity we have \S\ < 
k. The general assumption for almost sparsity is intuitively 
the fact that the total magnitude of the almost sparse terms 
should be small enough that so that it does not disturb the 
overall structure of the signal which may make the recovery 
impossible or very errornous. Since M — n ^ an d ^ e 

total contribution of the 'near-zero' elements is small we can 
assume that nA is small enough. We will use this assumption 
throughout this chapter. 

In order to make the analysis for almost fc-sparse signals 
simpler we will use a high quality expander graph which is 
right-regular as well 5 . The following lemma which is proved 
in [4] gives us a way to construct right-regular expanders 
from any expander graph without disturbing its characteristics 
(lemma 2.3 in [4]. 

Lemma 6 (right-regular expanders): From any left-regular 
(fc, 1 — e) unbalanced expander graph G with left size n, right 
size to, and left degree d it is possible to efficiently construct 
a left-right-regular (fc, 1 — e) unbalanced expander graph H 
with left size n, right size m' < 2m, left side degree d! < 2d, 
and right side degree D = [^] 

Corollary 1: There exists a (fc, 1 — e) left-right unbal- 
anced expander graph with left side size n, right side size 
to = O(fclogn), left side degree d = O(logn), right side 
degree D = O(^f^) = 0(1). Also based on the explicit 
constructions of expander graphs, explicit construction for 

5 the right-regularity assumption is just for the simplicity of the analysis 
and as we will discuss it is not mandatory. 



right-regular expander graphs exists as well. 

We will use the above right-regular high-quality expander 
graphs in order to perform robust signal recovery efficiently. 
The following algorithm generalizes the k — sparse recovery 
algorithm and can be used to find the position and sign of 
the fc largest elements of an almost fc-sparse signal x from 
y = Ax. Throughout the algorithm at each iteration t let p t — 
2tA+(D-t-l)\ and <j> t = 2tA+(D-t)\. where D = 0(n) 
is the right side degree of the expander graph. Throughout 
the algorithm we will assume that L > 2kA + DX. Hence 
the algorithm is appropriate for a family of almost fc-sparse 
signals for which the magnitude of the significant elements is 
large enough. We will assume that k is a small constant; when 
k is large with respect to n, (k — 0(n)), the (an, |) constant 
degree expander sketch proposed in [1] works pretty well. 

Algorithm 3 The 0(k) iteration, robust recovery algorithm 
to find the position and sign of the k largest elements of an 
almost-fc-sparse signal x and then a close fc-sparse approxi- 
mation for it. 

1) Initialize x = nx i. 

2) At each iteration t, if \y — Ailoo < 4>t determine the 
positions and signs of the significant components in x as 
the positions and signs of the non-zero signal components 
in x\ go to 5. 

3) ELSE find a variable node say Xj such that at least (1 — 
2e) d of the measurements it participate in are in either 
of the following categories: 

a) They have gaps which are of the same sign and have 
absolute values between L — A — A — p t and L + 
A + A + p t . Moreover, there exists a number G G 
{0, L + A, L - A} such that \y - A.x\ are all < <j> t 
over these (1 — 2e) d measurements if we change Xj 
to G. 

b) They have gaps which are of the same sign and have 
absolute values between 2L — 2A — p t and 2L+2A+ 
p t . Moreover, there exists a number G € {0,L + 
A, L — A} such that \y — A.x\ are all < <j> t over 
these (1 — 2e) d measurements if we change xj to 
G. 

set Xj <— G, and go to 2 for next iteration. 



elements from x contributes : 



4) 
5) 



pick the set of k significant elements of the candidate sig- 
nal x T . Let A'„,,, \ i be the sketch matrix A restricted 
to those elements, output the solution of the optimization 
problem : find a vector u to minimize \A'u — y\i- 



In order to prove the algorithm we need the following def- 
initions which are the generalization of the similar definitions 
in the exactly fc-sparse case. 

Definition 8: At each iteration t, we define S t an indicator 
of the difference between the estimate x and x : 



G t = {i\\Vi 



7 , 

3 = 1 



■j |oo 



> XD}. 



Theorem 8 (Validity of the algorithm 3): The first part of 
the algorithm will find the position and sign of the fc significant 
elements of the signal x (or more discussion see [1]). 

Proof: This is very similar to the proof of the validity of 
the exactly fc-sparse recovery algorithm. We will exploit the 
following facts. 

• x is almost fc — sparse so it has at most fc significant 
elements. Initially S\ = fc and G\ < kd. 

• Since at each iteration only one element Xj is selected, 
at each iteration t there are at most t elements Xj such 
that both Xj and Xj are in the significant level with the 
same sign. 

• If St < 2fc then S t +i < 2fc (Preservation Lemma), and by 
the neighborhood theorem at each round (1 — 2e)\S t \d < 
G t . 

• If St < 2k by the neighborhood theorem there exists 
a node Xj G St which is the unique node in St that 
is connected to at least (1 — 2e)d parity check nodes. 
This node is in S t . It differs from its actual value in the 
significance level or at sign. In the first case the part a) 
of the recovering algorithm will detect and fix it and in 
the second case the part b) of the algorithm will detect 
and fix it. For further discussion please refer to [1]. 



As a direct 
after T = 



result 

kd 



\G t+ i\ < \G t \ - (1 - 
iterations we will have 



(l-4e)d 

Consequently St = after at most 2fc iterations 



4e)d . 
Gt = 



So 
0. 



This means that after at most 2fc iterations the set St = 
{j\£j and Xjin different levels or with different signs.} will 
be empty and hence the position of the fc largest elements 
in xt will be the position of the fc largest elements in x. ■ 
Remark This algorithm like the exact fc-sparse counterpart 
needs at most 2fc iterations. Now by exploiting the simple 
structure of the adjacency matrix of the expander graph, (again 
in a similar manner to the exact fc-sparse case), since G\ < kd 
initially we only need to construct a binary search tree for kd 
elements. Moreover, at each iteration G t +i < G t . So even 
though at each iteration (1 — 2e)d nodes are deleted from the 
tree and at most 2ed possibly new nodes are added to the tree 
the size of the tree never increases. Hence each iteration can 
be implemented efficiently in 0(d\ogd) time complexity, and 
the algorithm will find the position of the fc largest elements of 
x in 0(k log n log log n) with a very small overhead. Note that 
the right-regularity assumption was only to make the analysis 
simpler and is not necessary. 

Knowing the position of the fc largest elements of x it will 
be easier to recover a good fc-sparse approximation. Based on 
the RIP—1 property of the expander graph we propose a way 



, ., „ , . , , , . , . to. recover a good approximation for a; in a time sub-linear in 

£>t = 1 1 \%i and Xjin different levels or large with different signs. \ ^ T .,-•„•, , • , • .• , 

^ 1 J J D n. We need the following lemma which is a direct result of 

Definition 9 (gap): At each iteration t, let G t be the set the RIP - 1 property of the expander graphs and is proved 

of measurement elements in which at least one 'significant' in [3] 



Lemma 7: Consider any u e R n such that \Au\i = b, and 
let S be any set of fc coordinates of u. Then we have: 

b 2e 



\us\i < 



d{l-2e) l-2e 



Mi- 



and: 



1 -2e 



\us\i < 



2e 



d(l - 2e) 1 - 2e 



Using Lemma 7 we will prove that the following minimization 
will recover a fc-sparse signal very close to the original signal: 
Theorem 9 (Final recovery): Suppose x is an almost fc- 
sparse signal and y = Ax is given where y £ R m and 
m = O(fclogn). Also suppose S is the set of the k largest 
elements of x. Now let A' be a submatrix of A containing 
columns from the positions of the k largest elements of x, so 
A' is a 0(k log n) x k dimensional matrix. Hence the following 
minimization problem can be solved in 0(poly(k, log n)) time 
complexity and will recover a fc-sparse signal v with very close 
norm-1 distance to the original x: 

min \A'v — j/| 2 

Proof: Suppose v is the recovered signal. Since v is fc- 
sparse we have Av — A'v and hence: 



\Av - Ax\!=\Av - y\! 

^A'v-yU 
<^m\A'v - y\ 2 
^^^rnlA'xs — y\ 2 
=V / m\Ax s - Ax\ 2 
<\fmXD\fm 
<V^XD 

=mDX = ndX. (12) 

The first two equations are only definitions. The third one 
is the Cauchy-Schwartz inequality. The fourth one is from the 
definition of v and the last one is due to the almost-fc-sparsity 
of x. Now by setting u — x — v in Lemma 7 and since v is 
fc-sparse and x is almost fc-sparse with the same positions, we 
will have: 



l-4e. \Ax - Av\i 

\us\i< 



2e 



1 - 2e' 



2e 



< 



d{l-2e) 1 

nX 2e 
+ — — 



< 



(1-26) 

nX 
■(l~2e) 
=0(nX). 



1 - 2e 

2e 
1 - 2e' 



nX 



As a result, since the signal is almost fc-sparse, the value of 
nX will be negligible and hence the recovered fc-sparse signal 
and the fc largest elements of the original signal will be very 
close to each other. So the result will be a fc-sparse signal 
approximating the original almost fc-sparse signal with very 
high precision. 



Remark: Recall that the right-regularity assumption is just 
for making the analysis simpler. As we mentioned before, it is 
not necessary for the first part of the algorithm. For the second 
part, it is used in the inequality \ Av — Ax\ < y/m\ Ax s — Ax\ 2 . 

However, denoting the i-th row of A by A i7 we have 



\Axs — Ax\ 2 = y/m 



Y^{A l {x s -x)) 2 < ^1 



\ i=i 



where Di denotes the number of ones in the i-th row of A. 
(In the right regular case, Di — D, for all i.) 
Therefore: 



\Axs — Ax\ 2 < \/mAy^ Dj = \pmXnd 

The only difference with the constant Di case is the extra 
m. But this does not affect the end result. 



VI. Conclusion 

In this paper we used a combinatorial structure called an 
expander graph, in order to perform deterministic efficient 
compressed sensing and recovery. We showed how using 
expander graphs one needs only O(fclogn) measurements and 
the recovery needs only O(fc) iterations. Also we showed how 
the expansion property of the expander graphs, guarantees the 
full recovery of the original signal. Since random graphs are 
with high probability expander graphs and it is very easy 
to generate random graphs, in many cases we might use 
random graphs instead. However, we showed that in cases that 
recovery guarantees are needed, just with a little penalty on 
the number of measurements and without affecting the number 
of iterations needed for recovery, one can use another family 
of expander graphs for which explicit constructions exists. We 
also compared our result with a very recent result by Indyk 
et al [22], and showed that our algorithm has advantages in 
terms of the number of required sketch measurements, the 
recovering complexity, and the simplicity of the algorithm in 
terms of the practical implementation. Finally, we showed how 
the algorithm can be modified to be robust. In order to do this 
we slightly modified the algorithm by using right-regular high 
quality expander graphs to find the position of the fc largest 
elements of an almost fc-sparse signal. Then exploiting the 
robustness of the RIP — 1 property of the expander graphs we 
showed how this information can be combined with efficient 
optimization methods to find a fc-sparse approximation for 
x very efficiently. However, in the almost fc-sparsity model 
that we used non-sparse components should have "almost 
equal" magnitudes. This is because of the assumption that 
L > fcA which restricts the degree of deviation for significant 
components. As a result, one important future work will be 
finding a robust algorithm based on more general assumptions. 
Table I compares our results with the previous papers. 



TABLE I 

Different Expander based recovery algorithms 



Paper 


R/D 


Explicit 


Sketch(m) 


# Iterations 


FullRecovery 


[15] 


R 


No 


O(fclogn) 


LP (0(n 3 )) 


Yes: RIP-2 


[3] 


D 


No 


fclog(f) 


LP (0(n 3 )) 


Yes:RIP-l 


[1] 
[2] 


D 
D 


Yes 
No 


9(n) 
0(k log n) 


O(k) 
0(k log n) 


Yes:RIP-l 
Yes: RIP-1 


Theorem:6 


D 


No 


o(fclog(f)) 
0(fe 2 (logn) 2 ) 


0(k) 


Yes:RIP-l 


Theorem:7 


D 


Yes 


O(k) 


Yes:RIP-l 
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